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DETAILED ACTION 

1 . This action is responsive to communications of application filed //2002. 

2. The disposition of the claims is as follows: claims 1-39 are pending in the application. 
Claims 1, 6, 11, 16, 20, 24 and 28-39 are independent claims. 

Claim Rejections - 35 USC § 102 

3. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed publication in this 
or a foreign country, before the invention thereof by the applicant for a patent. 

4. Claims 1-10, 28, 28, 34, 35 are rejected under 35 U.S.C. 102(a) as being disclosed by 
Chakraborty et al., (US Patent No. : 6,462,754 Bl), hereinafter Chakraborty. 

A. Chakraborty discloses claim 1, "A method of describing object region data about an 
object in a video over a plurality of frames [col. 3, Ins. 8-25; col. 5, Ins. 46-51; col. 10, Ins. 13- 
14], the method comprising: extracting an object from each of the frames [col. 7, Ins. 25-34, Ins. 
50-59]; approximating the object of each of the frames using one of predetermined figures 
defined by representative points for each of the frames[col. 3, Ins. 32-39]; extracting the 
representative points the figure for each of the frames, one of the representative points being a 
reference point represented by a coordinate value and one of the remaining representative points 
being represented by a relative position data with reference to the reference point [col. 4, Ins. 8- 
15, Ins. 43-46]; approximating a first trajectory with a first function, the first trajectory being 
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obtained by arranging, in the frames advancing direction, position data about the reference point 
[col. 4, Ins. 50-65; col. 8, In. 26 - col. 9, In. 67, wherein V x corresponds to "first trajectory with a 
first function"]; approximating a second trajectory with a second function, the second trajectory 
being obtained by arranging in the frames advancing direction, the relative position data about 
the one of the remaining points with reference to the reference point, wherein V y corresponds to 
"a second trajectory with a second function"; and describing the object region data using the first 
and second functions [col. 4, Ins. 50-65; col. 8, In. 26 - col. 9, In. 67]" as [detailed]. 

B. Chakraborty discloses claim 2, "The method according to claim 1, wherein said object 
region data comprises information representing a range of frames in which the object exists in 
the video data and information identifying the figure approximating the object region: supra for 
claim 1 , wherein [between frames] corresponds to "a range of frames". 

C. Chakraborty discloses claim 3, "The method according to claim 1, wherein said object 
region data comprises one of information representing related information linking to the object 
and information representing a method of accessing the related information" supra for claim 1 
and furthermore in [col. 3, Ins. 39-56]. 

D. Chakraborty discloses claim 4, "The method according to claim 1 , wherein said relative 
position data are components of differential vectors between the one of said the representative 
points and remaining of the representative points" supra for claim 1 and furthermore in [col. 8, 
Ins. 30-39]. 

E. Chakraborty discloses claim 5, "The method according to claim 1, wherein said object 
region data comprises parameters of the functions" supra for claims 1 and 4. 
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F. Per independent claim 6, this is directed to a method for performing the method of 
independent claim 1, and therefore is rejected to independent claim 1. 

G. Per dependent claims 7-10, these are directed to a method for performing the method of 
dependent claims 2-5, respectively, and therefore are rejected to dependent claims 2-5. 

H. Per independent claims 28, 29 and 34, 35, these are directed to a article of manufacture 
and computer data signal, respectively, for performing the method of independent claims 1 and 
6, respectively, and therefore are identically rejected to independent claims 1 and 6. 

Claim Rejections - 35 USC § 103 

5. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall hot be negatived by the 
manner in which the invention was made. 

6. Claims 1 1-27, 30-32 and 36-38 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Chakraborty as applied to claim 1-10, 28, 28, 34, 35, above, and further in view of Jasinschi 
et al, (US Patent Number 6,504,569 Bl), hereafter Jasinschi. 

A. Chakraborty discloses claim 1 1 , "A method of describing object region data about an 
object in video data over a plurality of frames, said the method comprising: extracting an object 
from each of the frames: approximating the object of each of the frames using one of 
predetermined figures defined by representative points for each of the frames; extracting the 
representative the points of the figure for each of the frames, one of the representative points 
being a reference point represented by a coordinate value and one of the remaining representative 
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points being represented by a relative position data with reference to the reference point; 
approximating a first trajectory with a first function, the first trajectory being obtained by 
arranging, in the frames advancing direction, data indicating a position of the reference point; 
and approximating a second trajectory with a second function, the second trajectory being 
obtained by arranging, in the frames advancing direction, the relative position data about the one 
of the remaining points with reference to the reference point; and describing the object region 
data using the first and second functions and depth information of the object" supra for claim 1 
and 6. 

However Chakraborty does not appear to disclose "describing the object region data 
using depth information of the object", but Jasinschi does in col. 1, Ins. 37-58 at "(10) 
Accordingly the present invention provides a method of generating 2-D extended images from 3- 
D data extracted from a video sequence representing a natural scene. In an image pre-processing 
stage image feature points are determined and subsequently tracked from frame to frame of the 
video sequence. In a structure-from-motion stage the image feature points are used to estimate 
three-dimensional object velocity and depth. Following these stages depth and motion 
information are post-processed to generate a dense three-dimensional depth map. World 
surfaces, corresponding to extended surfaces, are composed by integrating the three-dimensional 
depth map information."; in col. 2, Ins. 52-65; and col. 3, Ins. 33-36. 

Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to apply means for interpolating vertices of the objects between frames to 
define motions of the objects of interest so that the objects of interest are tracked disclosed by 
Chakraborty in combination with depth determining information disclosed by Jasinschi, and 
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motivated to combine the teachings because it would provide a method of generating 2-D 
extended images from 3-D data extracted from a two-dimensional video sequence as revealed by 
Jasinschi in col. 1, lines 32-34. 

B. Per dependent claims 12 and 13, these are directed to a method for performing the 
method of dependent claims 2 and 3, respectively, and therefore are rejected to claim 1 1 and to 
dependent claims 2 and 3. 

C. Per dependent claim 14, "The method according to claim 1 1, wherein said object region 
data is described by using the depth information of the object and parameters of the functions." is 
disclosed supra by Chakraborty for claim 4 and supra by Jasinschi for claim 1 1 . 

D. Per dependent claim 15, "The method according to claim 11, wherein said depth 
information is a relative depth and has a discrete level value" is disclosed supra by Chakraborty 
for claim 4 and supra by Jasinschi for claim 1 1 and in col. 7, Ins. 14-19 at "Step 4: Extract the 
camera rotation matrix R and the camera translation vector T from the computed essential matrix 

E. Step 5: Given R and T estimate the depth Z.sub.i at every feature point F.sup.i.sub.k. " 

E. Claim 16, "A method of describing object region data about an object in video data over 
a plurality of frames, the method comprising: extracting an object from each of the frames; 
approximating the object of each of the frames using one of predetermined figures defined by 
representative points for each of the frames; extracting the representative points of the figure for 
each of the frames, one of the representative points being a reference point represented by a 
coordinate value and one of the remaining representative points being represented by a relative 
position data with reference to the reference point; approximating a first trajectory with a first 
function, the first trajectory being obtained by arranging, in the frames advancing direction, data 
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indicating a position of the reference point; approximating a second trajectory with a second 
function, the second trajectory being obtained by arranging, in the frames advancing direction, 
the relative position data about the one of the remaining points with reference to the reference 
point; and describing the object region data using the first and second functions and display flag 
information indicating a range of frames in which the object or each of the representative points 
is visible or not." is disclosed by Chakraborty supra for claim 1 . However Chakraborty does not 
appear to disclose "display flag information indicating a range of frames in which the object or 
each of the representative points is visible or not", but Jasinschi does in col. 4, Ins. 20-28 at "The 
inputs to the 3-D camera parameter estimator 16 are raw video images, denoted by Lsub.k, and 
the corresponding "alpha" images, denoted by A.sub.k. The alpha image is a binary mask that 
determines the "valid" regions inside each image, i.e., the regions of interest or objects, as shown 
in FIG. 3 where FIG. 3A represents an image Lsub.k from a tennis match and FIG. 3B represents 
the alpha image A.sub.k for the background object with the tennis player blanked out." Wherein 
[binary mask] corresponds to "display flag information"; and [valid regions] corresponds to 
"object is visible or not". 

Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to apply object tracked disclosed by Chakraborty in combination with alpha 
images A.sub.k disclosed by Jasinschi, and motivated to combine the teachings because it would 
provide a method of generating 2-D extended images from 3-D data extracted from a two- 
dimensional video sequence as revealed by Jasinschi in col. 1 , lines 32-34. 
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F. Per dependent claims 17 and 18, these are directed to a method for performing the 
method of dependent claims 2 and 3, respectively, and therefore are rejected to claim 16 and to 
dependent claims 2 and 3. 

G. Per dependent claim 19, "The method according to claim 16, wherein said object region 
data is described by using the display flag information and parameters of the functions" is 
disclosed supra by Chakraborty and Jasinschi for claim 16 supra. Wherein alpha images A.sub.k 
corresponds to display flag information for valid regions. 

H. Claim 20, "A method of describing object region data about an object in video data over 
a plurality of frames, said method comprising: approximating the object using a figure for each 
of said frames; extracting a plurality of points representing the figure for each of said frames; 
approximating trajectories with functions, the trajectories being obtained by arranging, in the 
frames advancing direction, data indicating positions of said plurality of points; and describing 
the object region data using the functions and object passing range information indicating a range 
where the figure approximating the object exist over said plurality of frames" is disclosed by 
Chakraborty and Jasinschi supra for claims 1 1 and 16. 

However Chakraborty does not appear to disclose "describing the object region data 
using the functions and object passing range information indicating a range where the figure 
approximating the object exist over said plurality of frames.", but Jasinschi does in col. 4, Ins. 
20-28 at "The inputs to the 3-D camera parameter estimator 16 are raw video images, denoted by 

I. sub.k, and the corresponding "alpha" images, denoted by A.sub.k. The alpha image is a binary 
mask that determines the "valid" regions inside each image, i.e., the regions of interest or objects, 
as shown in FIG. 3 where FIG. 3 A represents an image I.sub.k from a tennis match and FIG. 3B 
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represents the alpha image A.sub.k for the background object with the tennis player blanked 
out." Wherein [object passing range information] corresponds to "sub.k"; and [valid regions] 
corresponds to "object exist". 

Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to apply object and passing range tracking disclosed by Lee in combination 
ranging information disclosed by Jasinschi, and motivated to combine the teachings because it 
would provide a method of generating 2-D extended images from 3-D data extracted from a two- 
dimensional video sequence as revealed by Jasinschi in col. 1, lines 32-34. 
I. Per dependent claims 21 and 22, these are directed to a method for performing the 
method of dependent claims 2 and 3, respectively, and therefore are rejected to claim 20 and to 
dependent claims 2 and 3. 

J. Per dependent claim 23, "The method according to claim 20, wherein said object region 
data is described by using the object passing range information and parameters of the functions." 
is disclosed supra by Chakraborty and Jasinschi for claim 20 supra and exemplified by 
Chakraborty. 

K. Per independent claims 30-32 and 36-38, these are directed to a article of manufacture 
and computer data signal, respectively, for performing the method of independent claims 11,16, 
and 20, respectively, and therefore are identically rejected to independent claims 11,16, and 20. 

7. Claims 24-27, 33 and 39 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Chakraborty et al., (US Patent 6,462,754 Bl), as applied to claims 1-5 above, and further in view 
of "Panoramic Image Mosaics", Heung-Yeung Shum, hereafter Shum. 
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A. Claim 24, "A method of describing object region data about an object moving in a 
panorama image formed by combining a plurality of frames with being overlapped, said the 
method comprising: extracting an object from each of the frames: approximating the object of 
each of the frames in the panorama image using one of predetermined figures defined by 
representative points for each of the frames; extracting the representative points of the figure in a 
coordinate system of the panorama image, one of the representative points being a reference 
point represented by a coordinate value and one of the remaining representative points being 
represented by a relative position data with reference to the reference point; approximating a first 
trajectory with a first function, the first trajectory being obtained by arranging, in the frames 
advancing direction, data indicating a position of the reference point; and approximating a 
second trajectory with a second function, the second trajectory being obtained by arranging, in 
the frames advancing direction the relative position data about the one of the remaining points 
with reference to the reference point: and describing the object region data using the first and 
second functions" is disclosed by Chakraborty supra for claim 1 . However Chakraborty does not 
appear to disclose "approximating the object of each of the frames in the panorama image using 
one of predetermined figures defined by representative points for each of the frames; extracting 
the representative points of the figure in a coordinate system of the panorama image, one of the 
representative points being a reference point represented by a coordinate value and one of the 
remaining representative points being represented by a relative position data with reference to 
the reference point", but Shum does in abstract and last paragraph of p. 2, at "This paper presents 
some techniques for constructing panoramic image mosaics from sequences of images. Our 
mosaic representation associates a transformation matrix with each input image, rather than 
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explicitly projecting all of the images onto a common surface (e.g., a cylinder). In particular, to 
construct a full view panorama, we introduce a rotational mosaic representation that associates a 
rotation matrix (and optionally a focal length) with each input image. A patch-based alignment 
algorithm is developed to quickly align two images given motion models. Techniques for 
estimating and refining camera focal lengths are also presented. 

In order to reduce accumulated registration errors, we apply global alignment (block adjustment) 
to the whole sequence of images, which results in an optimally registered image mosaic. To 
compensate for small amounts of motion parallax introduced by translations of the camera and 
other unmodeled distortions, we develop a local alignment (deghosting) technique which warps 
each image based on the results of pairwise local image registrations. By combining both global 
and local alignment, we significantly improve the quality of our image mosaics, thereby enabling 
the creation of full view panoramic mosaics with hand-held cameras. 
We also present an inverse texture mapping algorithm for efficiently extracting environment 
maps from our panoramic image mosaics. By mapping the mosaic onto an arbitrary texture- 
mapped polyhedron surrounding the origin, we can explore the virtual environment using 
standard 3D graphics viewers and hardware without requiring special-purpose players. 
Third, any deviations from the pure parallax-free motion model or ideal pinhole (projective) 
camera model may result in local misregistrations, which are visible as a loss of detail or 
multiple images (ghosting). To overcome this problem, we compute local motion estimates 
(block-based optical flow) between pairs of overlapping images, and use these estimates to warp 
each input image so as to reduce the misregistration. 
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Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to apply object tracking disclosed by Chakraborty in combination with 
panorama image mosaics disclosed by Shum, and motivated to combine the teachings because it 
would provide a technique for constructing panoramic image mosaics from sequences of images 
as disclosed by Shum in abstract. 

B. Per dependent claims 25-27, these are directed to a method for performing the method of 
dependent claims 2, 3, and 5, respectively, and therefore are rejected to claim 24 and to 
dependent claims 2, 3, and 5. 

C. Per independent claim 33 and 39, these are directed to an article of manufacture and 
computer data signal, respectively, for performing the method of independent claim 24 and 
therefore are identically rejected to independent claim 24. 

Response to Arguments 

8. Applicant's arguments with respect to claiml-39 have been considered but are moot in 
view of the new ground(s) of rejection. 

Responses 

9. Responses to this action should be mailed to: Commissioner of Patents and Trademarks, 
Washington, D.C. 20231. If applicant desires to fax a response, (703) 872-9306 may be used for 
formal communications. 

Hand-delivered responses should be brought to Crystal Park II, 2121 Crystal Drive, 
Arlington, VA., Sixth Floor (Receptionist). 
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Inquiries 



10. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Greg Cunningham whose telephone number is (703) 308-6109. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Matthew Bella, can be reached on (703) 308-6829. 

Any response to this action should be mailed to: 



Hand-delivered responses should be brought to Crystal Park II, 2121 Crystal Drive, 
Arlington, VA, Sixth Floor (Receptionist). 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the Technology Center 2600 Customer Service Office whose telephone 
number is (703) 306-0377. 



Commissioner of Patents and Trademarks 



Washington, D.C. 20231 



or faxed to: 



(703) 872-9306 (for Technology Center 2600 only) 
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